Overview of Popular Benchmark Sets

ONE OF THE MOST DIFFICULT TASKS CAD users face is the evaluation and comparison of different tools and algorithms. For commercial software purchasers, it is vital to understand how well a given tool does the required job and which of many possible choices is best for the kinds of problems a user will face. For the tool developer, whether in academia or industry, the efficiency of critical algorithms must be measured and compared to understand both tool behavior and progress over time. Over the years, there have been many attempts to create and use neutral benchmarks for tool evaluation and comparison. Typically, a benchmark set consists of a collection of circuits in a common format, which attempt to represent a range of problems for evaluating algorithms and tools within an important problem domain. In principle, if everyone uses the same test cases to evaluate similar tools, it should be straightforward to compare results— although this is rarely true in reality. When using benchmark sets, or interpreting results, keep the following things in mind: I Few real tools are as simple as the examples studied in algorithms courses. For instance, a sort algorithm does one specific job; its complexity may be calculated rigorously, and its real-world performance can easily be measured and compared to other algorithms designed for the same task. But what about a wire-routing algorithm? Here, there are also core algorithms whose theoretical complexity may be calculated, but in addition , there is an interplay of algorithms and constraints that blurs the picture. Theoretical complexity or benchmark performance of an algorithm often says little about the performance of an entire tool or flow. I It is a natural human tendency to choose test cases that demonstrate the good qualities of our tools, and to avoid those that do not. Beware of published results that cite a benchmark set, but do not report results for all members of the set. I Do not believe reported results until you verify the results by repeating the test. In many cases, there will be unspoken assumptions and boundary conditions that make it impossible for another party to achieve published results. I How do you know that the test case truly represents your problem? In most cases, real designs are not publicly used to evaluate tools because of proprietary concerns. Benchmark sets are surrogate circuits chosen to represent the kinds of problems a tool …